Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

init apache hadoop package #29037

Closed
wants to merge 33 commits into from
Closed

init apache hadoop package #29037

wants to merge 33 commits into from

Conversation

kranurag7
Copy link
Contributor

Fixes:

Related:

Pre-review Checklist

For new package PRs only

  • This PR is marked as fixing a pre-existing package request bug
    • Alternatively, the PR is marked as related to a pre-existing package request bug, such as a dependency
  • REQUIRED - The package is available under an OSI-approved or FSF-approved license
  • REQUIRED - The version of the package is still receiving security updates
  • This PR links to the upstream project's support policy (e.g. endoflife.date)

@kranurag7 kranurag7 marked this pull request as ready for review September 24, 2024 15:25
EyeCantCU
EyeCantCU requested changes Sep 24, 2024
View reviewed changes
hadoop.yaml Outdated Show resolved Hide resolved
hadoop.yaml Outdated Show resolved Hide resolved
hadoop.yaml Outdated Show resolved Hide resolved
hadoop.yaml Outdated Show resolved Hide resolved
hadoop.yaml Outdated Show resolved Hide resolved
hadoop.yaml Outdated Show resolved Hide resolved
hadoop/pombump-properties.yaml Outdated Show resolved Hide resolved
EyeCantCU
EyeCantCU reviewed Sep 24, 2024
View reviewed changes
hadoop.yaml Outdated Show resolved Hide resolved
@kranurag7 kranurag7 mentioned this pull request Sep 25, 2024
Merged
EyeCantCU pushed a commit that referenced this pull request Sep 26, 2024
@kranurag7
add py3-robotframework package (#29331)
97603d7 
required deps for [hadoop
package](#29037).

ref:
https://github.com/apache/hadoop/blob/5bb6f450340fa3deb931571d08d42a3f358af8bf/Dockerfile#L33

---------

Signed-off-by: kranurag7 <[email protected]>
@kranurag7 kranurag7 marked this pull request as draft September 27, 2024 12:17
@kranurag7
init apache hadoop package
0200c54 
Signed-off-by: kranurag7 <[email protected]>
@kranurag7
CVEs fix
06112f2 
Signed-off-by: kranurag7 <[email protected]>
@kranurag7
pass in properties via properties-file
3733dc9 
Signed-off-by: kranurag7 <[email protected]>
@kranurag7
bump deps
743b31f 
this commit bump the deps and also comments out the ones that are not
compiling.

Signed-off-by: kranurag7 <[email protected]>
@kranurag7
add nodejs and yarn
467735f 
for some reason build on x86 is working good but not for arch so adding
these two as build dependencies. This is required because we are
building yarn-ui.

Signed-off-by: kranurag7 <[email protected]>
@kranurag7
build only for x86 arch
2cbc07f 
frontend-maven-plugin which is required for building the UI doesn't work
on aarch64 architecture so we are only building for x86 arch.

Signed-off-by: kranurag7 <[email protected]>
@kranurag7
some more CVE fixes
29604a7 
Signed-off-by: kranurag7 <[email protected]>
@kranurag7
address review comments
b37dd89 
Signed-off-by: kranurag7 <[email protected]>
@kranurag7
bash is required to build this package
ad38993 
Signed-off-by: kranurag7 <[email protected]>
@kranurag7
remove versioned directory
3c3ffa4 
Signed-off-by: kranurag7 <[email protected]>
@kranurag7
add hadoop-runner subpackage
eb50787 
Signed-off-by: kranurag7 <[email protected]>
@kranurag7
add more runtime deps
09597fb 
Signed-off-by: kranurag7 <[email protected]>
@kranurag7
replace nmap with netcat-openbsd and add coreutils
ac61518 
we need coreutils because we are missing env binary on latest image.

Signed-off-by: kranurag7 <[email protected]>
@kranurag7
add yarn as runtime deps to hadoop
706f5ba 
Signed-off-by: kranurag7 <[email protected]>
@kranurag7
remove yarn again and fix symlink
48e73f0 
Signed-off-by: kranurag7 <[email protected]>
@kranurag7
add logging in destdir
1582a8d 
Signed-off-by: kranurag7 <[email protected]>
@kranurag7
add curl as runtime dependency to image
5cad337 
Signed-off-by: kranurag7 <[email protected]>
@kranurag7
symlink profiler as well
6e25bfe 
Signed-off-by: kranurag7 <[email protected]>
@kranurag7 kranurag7 marked this pull request as ready for review October 4, 2024 19:50
EyeCantCU
EyeCantCU requested changes Oct 4, 2024
View reviewed changes
hadoop.yaml Outdated Show resolved Hide resolved
hadoop.yaml Outdated Show resolved Hide resolved
kranurag7 and others added 9 commits October 6, 2024 13:12
@kranurag7
add more build time deps to compile natively (still failing)
91870d7 
Signed-off-by: kranurag7 <[email protected]>
@EyeCantCU
fix(hadoop): Fix build with OpenSSL > 1.1.1
5fdef78 
These are undefined when using OpenSSL > 1.1.1 so the build fails.
Remove the version resitriction so that the common client builds

Signed-off-by: RJ Sampson <[email protected]>
@EyeCantCU
chore(hadoop): Build with snappy
572a2f9 
Signed-off-by: RJ Sampson <[email protected]>
@EyeCantCU
chore(hadoop): Build with FUSE support for HDFS
233b8d4 
Signed-off-by: RJ Sampson <[email protected]>
@kranurag7
use gnutar and yam hadoop.yaml
eaee7c9 
using gnutar because busybox tar is missing some flags.

Signed-off-by: kranurag7 <[email protected]>
@kranurag7
add abseil-cpp-dev libraries for native build as well
8108c69 
Signed-off-by: kranurag7 <[email protected]>
@kranurag7
add patch to fix the build and more deps
ed5829c 
Signed-off-by: kranurag7 <[email protected]>
@kranurag7
add find and awk as runtime dependencies
42a7e52 
some example jobs require this so adding this as runtime dependency.

Signed-off-by: kranurag7 <[email protected]>
@kranurag7
some more build time libraries
1d5129c 
these are some additional libraries being added which are optional at
build time but hadoop checknative -a command flags mentions it so adding
it here.

Signed-off-by: kranurag7 <[email protected]>
EyeCantCU
EyeCantCU requested changes Oct 8, 2024
View reviewed changes
hadoop.yaml Outdated Show resolved Hide resolved
hadoop.yaml Outdated Show resolved Hide resolved
hadoop.yaml Outdated Show resolved Hide resolved
hadoop.yaml Outdated Show resolved Hide resolved
hadoop.yaml Outdated Show resolved Hide resolved
hadoop/fix-build-openssl.patch Outdated Show resolved Hide resolved
@kranurag7 kranurag7 mentioned this pull request Oct 8, 2024
Closed
@kranurag7
Copy link
Contributor Author

CI is failing because hadoop-runner is not there which was include as runtime dependency to this package.

Logs:

failed to bundle: unable to build graph: hadoop-3.4-3.4.0-r0: unable to resolve dependency hadoop-runner-3.4: could not find package that provides hadoop-runner-3.4 in indexes

@kranurag7
Copy link
Contributor Author

closing given this is not longer tracked here.

@kranurag7 kranurag7 closed this Oct 9, 2024
@kranurag7 kranurag7 deleted the kr/apache-hadoop branch October 9, 2024 09:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@EyeCantCU EyeCantCU EyeCantCU requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kranurag7 @EyeCantCU